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ABSTRACT 

This paper introduces an evaluation tool prepared to examine the quality of visuals in course-books. The tool is 
based on Mayer’s Cognitive Theory of Multimedia Learning (i.e. Generative Theory) and its principles 
regarding the correct use of illustrations within text. The reason to generate the tool, the development process 
along with the theoretical procedures followed, reliability studies and users’ manual of the instrument are 
provided within the study. 
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INTRODUCTION 

The essence of the current century brings about an overwhelming amount of information which involves using 
efficient mechanisms to ameliorate learning and teaching activities. Thus, presenting learning materials merely 
through textual information may not lead to efficient learning demanded by the excessive amount of information. 

Course-books serve as an important segment of the teaching-learning process. Instructional procedures have 
been supported with a variety of illustrations since Thorndike’s (1912) recommendation that pictures were an 
effective device in instruction. In recent decades, more importance has been given to pictures used in course- 
books. Salomon (1979) named pictures and other supplementary materials to support instruction as symbol 
systems. He maintained that media without symbol systems were just like mathematics without numbers. 
Therefore, visuals that are generated with the aim of decorating, describing, defining, explaining, clarifying and 
organizing learning units play an important role in supporting instruction (Mayer, 2000). 

Previous studies revealed that using visuals along with textual information enhanced learners’ performance to a 
considerable extent (Chun & Plass, 1996; Levin, 1981; Mayer & Moreno, 2003; Mayer & Sims, 1994; Mousavi, 
Low, & Sweller, 1995; Seghayer, 2001). According to the Multimedia Principle of Mayer (2000), when words 
and pictures are presented simultaneously, learners have a chance to construct both verbal and pictorial mental 
schemas, and build connections between them. Within the scope of the current study, many forms of visuals are 
taken into account including pictures, flowcharts, diagrams, illustrations and maps. 

Multimedia Learning 

According to Kommers, Grabinger and Dunlap (1996), multimedia refers to computer-based applications where 
users are provided with information through different types of media. Mayer (2000) provides a more specific 
definition which is in line with the focus of the current study. He defines multimedia as the presentation of the 
learning material using both words and pictures. By words, he means that the material is presented in verbal form, 
such as using spoken or printed text. By pictures, he means that the material is presented in pictorial form, such as 
using illustrations, graphs, photos, or maps. Mayer (2000) claims that his definition is comprehensive enough to 
cover all other definitions and scenarios involving the integration of textual material with visuals. The current 
study follows the definition of Mayer while developing the evaluation tool. 

The theoretical grounds for learning in multimedia environments have been built within the frameworks of the 
Dual Coding Theory (Paivio, 1986), Cognitive Load Theory (Chandler & Sweller, 1991) and Generative Theory 
(Mayer, 1997). The Dual Coding Theory maintains that cognition is served by two interdependent systems one of 
which is specialized for dealing with verbal information (i.e. text and speech), while the other one processes non- 
verbal information (i.e. graphics and animation). By interdependent, Paivio (1986) means that those systems are 
interconnected, but can function independently. The theory focuses on how the material is represented in human 
memory and cognition rather than how it is perceived by the learner. 

The Cognitive Load Theory focuses on the learners’ cognitive effort employed during learning. The basic 
premise of the theory is that it is easier to learn new information and automatize it if the teaching method can 
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reduce the demands on learners’ working memory (Chandler & Sweller, 1991; Mousavi et al., 1995). Sweller, 
Van Merrienboer and Paas (1998) provide a scrutinized description of the theory. They state that the theory 
assumes a limited capacity working memory that involves partially independent components to deal with verbal 
and non-verbal material. On the other hand, it assumes a relatively unlimited long-term memory, which includes 
billions of schemas varying in degrees of automation (i.e., automatic processing of information). It is possible to 
store seven items at a time in the working memory three of which remaining unprocessed whilst four of which 
were processed for approximately twenty seconds (Van Merrienboer & Sweller, 2005). However, the long-term 
memory is almost limitless. Thus, effective transfer of the information from the bottle neck of the learning (i.e. 
short-term memory) to the long-term memory is the center of the theory (Gerjets & Scheiter, 2003). The 
implicational purpose of the theory is to design a variety of novel instructional materials following the 
assumption that the short-term memory load should be minimized and schema construction should be 
encouraged. The theory maintains that the working memory load can be affected by either the intrinsic nature of 
the material (i.e. intrinsic cognitive load), or by the manner in which the material is presented, or the types 
activities students were exposed to (i.e. extraneous cognitive load). Intrinsic cognitive load cannot be modified 
through instructional interventions because it is intrinsic to the material being dealt with. On the other hand, 
extraneous cognitive load is the unnecessary cognitive load and can be modified through instructional designs. 
Finally, germane cognitive load is defined, which refers to the effort that ameliorates the constitution of schemas. 
Like extraneous cognitive load and unlike intrinsic cognitive load, germane cognitive load is influenced by the 
instructional design. The manner in which the information is presented to learners and learning exercises required 
of learners determine the level of the germane cognitive load. Unlike extraneous cognitive load, germane 
cognitive load does not interfere with learning, rather it enhances learning (Brtinken, Plass, & Leutner, 2003; 
Paas, Renkl, & Sweller, 2003). More specifically, if memory resources are being used to engage in search, 
scanning or skimming for details, extraneous cognitive load is maximized, whereas the germane cognitive load is 
maximized through engaging learners’ resources to schema acquisition and automation through informed 
exercises. In brief, good instructional designs reduce extraneous load and increase germane load (Sweller et al., 
1998). 

The Generative Theory of multimedia learning (Mayer, 1997 & 2001) integrates above theories in a way that 
focuses on presenting information in dual mode without increasing the cognitive load. Mayer and Moreno (2003) 
summarize the principle of the Generative Theory, which are dual-channel assumption, limited-capacity 
assumption and active processing assumption. As the names suggest, the dual-channels assumption is based on 
the Dual Coding Theory of Paivio (1986) claiming that human beings possess two separate but interdependent 
systems for processing verbal and pictorial material. The limited-capacity assumption is based on the Cognitive 
Load Theory of Chandler and Sweller (1991) maintaining that each channel is limited in the amount of material 
that can be processed by learners at one time. The final assumption, active processing, states that meaningful 
learning involves the cognitive process of actively building connections between verbal and pictorial 
representations. Thus, the learners engage in cognitive activities of selecting, organizing and integrating 
knowledge (see Figure 1). 
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Figure 1. Mayer’s Generative Theory of Multimedia Learning 
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In brief, the Generative Theory suggests that students learn better from words and pictures than from words 
alone, since dual presentation of information help students construct verbal and pictorial mental models together 
and build connections between these models rather than constructing either a verbal or a pictorial mental model 
(Mayer, 2000). Within the framework of the theory, Mayer (1997 & 2000) introduces several design principles 
and revises those principles in Mayer and Moreno (2002). The principles relevant to current study are explained 
as follows. The multimedia representation principle states that it is better to present a learning material in words 
and pictures than merely in words. The contiguity principle states that it is better to present corresponding words 
and pictures simultaneously rather than separately when presenting a learning material. Finally, the coherence 
principle states that multimedia explanations are better understood when they include very few extraneous words 
(P-107). 

Studies focusing on cognitive processing of textual and visual information yielded several principles for 
designing and using multimedia to minimize the cognitive load and to help learners receive and automatize new 
knowledge. Like any other scientific field, the field of instructional design and technology suffers from the 
colossal gap between the theory and practice. That is, the design principles derived from the grounded theory are 
not applied in instructional activities and learning materials. Mayer (2000) claims that most of the visuals used in 
instructional materials do not have an instructional purpose, rather they serve as decorative items. Besides, most 
visuals are likely to increase the extraneous cognitive load since they are used haphazardly in most course-books 
rather than in an informed way. 

As the literature indicates, seeking for germane cognitive load in instructional materials facilitates realization of 
instructional objectives. Thus, development of course-books in a way to reduce the extraneous cognitive load 
and to increase the germane cognitive load should encourage efficient learning. In this respect, visuals constitute 
a major device regarding cognitive load. To our knowledge, course-books have not yet been comprehensively 
evaluated in terms of the quality of visuals regarding the cognitive load, even thought tens of visuals are 
provided in each course -book. This problem anticipates the necessity to create an instrument to evaluate the 
course-books in terms of the cognitive quality of visuals used. 

E§gi (2005) suggests that course-book evaluation should concentrate on two aspects. The first one is the content 
analysis, which has been investigated in the literature through preparing content checklists or asking for expert 
opinions. The second one is the evaluation of visuals, which has not been conducted with a quantitatively robust 
instrument so far. Alpan (2004) states both types of evaluation have been neglected. While graphic experts focus 
on the face validity and ignore the content, educational researchers focus on the content and ignore the face 
validity. 

Kaptan and Kaptan (2005) put course-book evaluation on the research agenda and maintain that the basic 
knowledge resources of our educational system, course-books, have problems in terms of their content validity 
and instructional design. They claim that no matter how much the authors have expertise in the subject area, can 
the course-books be of bad quality if the instructional design principles are not followed. These principles 
involve a large variety of issues including content, cover design, typography, paper and press quality, quality of 
the visuals, and so on. 

Needs analysis and determining problems to be solved are the first steps in instructional design (Morrison, Ross, 
& Kemp, 2004). Thus, the current evaluation apparatus should first focus on the current instructional materials 
and designate the imperfections if there are any. Based on data collected through evaluation of currently used 
materials, necessary steps can be taken. Those steps can either be the modification of the current materials or the 
substitution of them with new ones that are designed in line with the grounded theory. Finally, course-book 
writers can make use of the apparatus in order to evaluate the materials they develop in terms of their suitability 
with the theory based on research. The current study seeks to develop an evaluation tool that is applicable to 
current materials. In this respect, it delves into the characteristics of the instrument to ameliorate it through 
expert opinion, and scrutinizes on reliability indices among several observers who have expertise in the field. 

Purpose 

The importance of the design and evaluation of educational materials grows constantly, since it is necessary to 
ensure the quality of course -books used in the teaching-learning process. The current study provides an 
evaluation tool to assist instructional design professionals in the assessment of course-books. The purpose of 
developing such an instrument could be summarized as follows: 

• to confirm whether the visuals used in course-books are suitable in terms of the Cognitive Theory of 
Multimedia Learning, 

• to guide course-book authors in creating course materials that contain better visuals. 
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• to evaluate some of currently used course-books to prepare a quality inventory of the current materials. 
Contribution of the Study 

To our knowledge, there is a lack of quantitatively interpretable and robust instruments measuring the quality of 
visuals in instructional materials regarding the Cognitive Theory of Multimedia Learning, which is a 
contemporary theory based on research. Thus, it is hoped that the evaluation tool developed within the scope of 
the current study is going to help instructional designers to assess instructional materials in a more robust way. 

METHODS AND PROCEDURES 

While developing the evaluation tool, the following steps were taken successively: 

1. Evaluation principles were determined. The principles are based on the Cognitive Theory of 
Multimedia Learning with an emphasis on reducing the extraneous cognitive load and increasing the germane 
cognitive load through instructional design interventions. The design principles proposed by Levin and Mayer 
(1993) were particularly helpful. 

2. Operational definitions regarding these principles were prepared. It is necessary to create a directive 
evaluation tool with clear definitions, descriptions and evaluations so that the tool can be implemented 
objectively and efficiently by different users. 

3. Expert opinion was sought for to sustain content, construct, and face validity of the instrument. 
Twenty academicians and graduate students studying at an instructional design and technology department 
(henceforth, IDT) in Turkey examined the tool, and asked for modifications for the ambiguous aspects of the 
users’ manual. They confirmed that the instrument could be useful after the modifications they suggested were 
done. Operational definitions of the criteria to evaluate visuals were improved after their suggestions. 

4. The internal consistency reliability of the instrument was sustained. Two units of a randomly selected 
computer course-book for K-8 students were evaluated by 6 academicians and 8 graduate students at an IDT 
department in Turkey. Grades each picture got were entered into SPSS 14.0 for windows to investigate the 
internal consistency of the instrument (i.e., Cronbach’s Alpha). 

Evaluation Criteria and Operational Definitions 

As mentioned before, most visuals used in course-books unfortunately do not have an instructional purpose 
(Mayer, 2000). Thus, the current instrument first asks the users to classify the visuals given in a course-book in 
terms of its type (See Appendix). Mayer (2000) categorizes each visual in an instructional material as belonging 
to one of the following categories: Decorative, representational, organizational and explanative. Decorative 
illustrations are aimed to interest or entertain the reader but do not improve the instructional message of the text. 
A picture depicting a group of students sitting and smiling in the classroom for a lesson on science and 
technology can be considered as a decorative illustration. Representational illustrations portray a single element 
at a time and provide the exact visual description of the material being taught. Thus representational illustrations 
answer the question ‘What is it like?’ A picture of the monitor with the heading “monitor” provided along with 
the text explaining monitor is a representational illustration. Organizational illustrations demonstrate relations 
among elements, such as maps or charts showing the main steps of the instructional design process or the main 
parts of a tutorial. They answer the question ‘How are they organized, sequenced, or branched?’ Finally, 
explanative illustrations explain how a system works, such as frames to explain how the hard disk stores 
information. They answer the question ‘How does it work?’ 

Mayer (2000) adapts several general design principles form Levin and Mayer’s study conducted in 1993. These 
principles were mostly applied to annotated illustrations. Within the context of the current study, the principles 
will be adapted so that they could be used for any visual in instructional materials. The names, operational 
definitions and contents of the criteria are provided below: 

1. Concentrated: The key ideas of the subject-matter should be highlighted both in the illustrations and 
in the text through using underlining styles, appropriate font sizes, font colors and styles, highlights, and arrows. 

2. Concise: In order to prevent redundancy, extraneous descriptions should be minimized in the text 
along with extraneous visual features such as unneeded colors and details. 

3. Correspondent: Corresponding illustrations and text fragments should be presented near each other 
on the very same page. 

4. Concrete: The text and corresponding illustrations should be delivered in ways that help learners 
visualize the learning material easily. 

5. Coherent: The presented texts and pictures should follow a consistent, clear and coherent structure. 
Thus segments explained in the text should be explained similarly in the picture. The text and the picture should 
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have the same organizational structure. Finally, the structure followed in the text, say cause-effect, should be 
supported with the illustrations as well. 

6. Comprehensible: The text and illustrations should be familiar to learners so that they could apply 
relevant past experiences to understand the material. 

7. Codable: Key terms in the text and key features of the visuals should be used consistently in ways 
that can make them more memorable. 

Evaluation Tool 

The chart provided in the Appendix allots a column for a single visual used in a given course-book. The upper 
row for the learning units is left empty since it is in a dynamic structure. More specifically, learning units are 
filled in by the users. These cells are designed as dynamic, since the number of pictures in each learning unit can 
change from material to material. While determining the learning units, the table of contents should be used so 
that the instrument is applied consistently by different evaluators. After the learning units are determined, picture 
names are given as ID numbers consecutively. Next, the necessity of the picture is stated in a dichotomous 
fashion, say yes-no. 

The type of the picture is marked with the initials of decorative, representational, organizational and explanative. 
In the current study, decisions of the observers in terms of the type of visual were entered into SPSS, and the 
Cronbach's Alpha was calculated. The alpha ( CC =0.8740) was higher than the suggested ideal alpha (i.e., .70) 
(Pallant, 2001). This indicates that observers interpreted the operational definitions of picture types consistently. 

After the process of determining the picture type is over, each picture is given a score in terms of the criteria 
whose name is provided on the left-side of the instrument. Whenever a picture completely meets a criterion, it 
gets two points on that criterion. If it meets the criterion, but still has some problems, it is given one point. 
Finally, if the picture does not meet the criterion, it is given zero point. In this respect, a perfect picture meeting 
7 criteria gets a total score of fourteen. The Cronbach's Alpha for the evaluations of our observers in this part 
was very high again ( a =0.8922). 

When the total scores are examined, pictures with 10 to 14 points are considered usable or appropriate, those 
with 6 to 9 points are considered usable if some modifications are realized, and finally pictures with 0 to 5 points 
are regarded useless, which should be removed from the instructional material. The tool calculates the averages 
of decorative, representational, organizational and explanative pictures in terms of each criterion. Thus, when 
several course-books are evaluated, it will be possible to make generalizations such as ‘organizational pictures 
are the most concentrated ones’ or ‘representational pictures generally lack correspondence’. 

At the right side of the table, the average scores of the material out of 100 regarding the criteria are given. More 
specifically, the material is given a concentrate, conciseness, correspondence, concreteness, coherence, codable 
and comprehensiveness grade out of 100. In this respect, 70 to 100 points mean that the material is appropriate in 
terms of the given criterion, 50 to 70 points mean that the material should be revised in terms of that criterion, 
and finally below 50 points mean that the material is not suitable in terms of the given criterion. Finally, the 
material gives a total grade to the book out of 100. This score is interpreted just like the criterion scores. The 
original table is prepared through MS Excel. It calculates all above points and provides the comments regarding 
those points on its own. 

At the bottom of the evaluation tool, extra lines are provided so that the user can add additional comments about 
the problematic pictures. This is deliberately done so that qualitative data can be blended with the quantitative 
information to get more efficient evaluation results. The users are asked to add extra comments whenever they 
give zero to a specific picture. Comments of observers were evaluated, and a consensus among them on the 
problematic pictures was observed in the pilot application. 

CONCLUSION 

The evaluation tool presented here can pose some limitations since the standardization of the instrument for a 
large variety of educational materials has not been realized yet. The instrument can be used in other course- 
books from different branches and levels to generalize findings to a larger variety of course-books. The tool 
could be considered a first step for further evaluative purposes. Formative evaluation of the instrument by 
students, teachers and instructional designers is also necessary, so that, the instrument can serve better each time 
it is used. 

In Turkey, the concepts and principles of instructional technology were taught as an integral part of the 
undergraduate programs in education within the context of the required course ‘Instructional Technologies and 
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Material Development'. Within the scope of this course, students are equipped with basic information 
technology skills along with selection, development, implementation and evaluation of media according to 
learning theories. Resources regarding the development of educational materials generally provide learners with 
the basics of materials development following the contents of the course, which were pre-determined by the 
Turkish Council of Higher Education (i.e., YOK). These resources have contributed to the advancement of 
instructional design and technology in Turkey. Moreover, they have also offered sound criteria to evaluate 
instructional materials (Demirel, Seferoglu, & Yagci, 2001; i§man, 2005; Kaya, 2005; §ahin & Yildinm, 1999; 
Yalin, 2004). However; generating an evaluation tool for visuals in course-books with a quantitatively robust 
instrument has not been a research focus so far, since the preliminaries of IDT have not been totally covered in 
Turkey yet. 

The instrument developed within the present study brings a new dimension to course-book evaluation in terms of 
the evaluation of visuals. Opinions of the experts along with the reliability indices suggest that the instrument is 
appropriate for the purpose of evaluating visual supports provided in course-books. Thus, the instrument can be 
used to evaluate course-books in terms of their concordance with the multimedia learning principles based on the 
Cognitive Theory of Multimedia Learning. Besides, the tool can be used to design new books that are congruous 
with the cognitive theory. 

The material focuses on the second type of evaluation mentioned by E§gi (2005) that is evaluating visuals, rather 
than the content. By generating this tool, authors try to bridge the gap between theory-based design principles 
and current applications to some extent, which was criticized before (Alpan, 2004; Kaplan & Kaplan, 2005). The 
instrument involves basic evaluation issues listed by Kaplan and Kaplan (2005). 

The material is also expected to give course-book authors in different disciplines ideas about the quality of their 
works. Moreover, the instrument is hoped to help designers in choosing and adapting pictures in accordance with 
the theory-based principles during materials preparation process. Further research should be conducted to 
enhance the current instrument by evaluating several course-books from different disciplines. 
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APPENDIX EVALUATION SCALE FOR VISUALS 
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